Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(autoware_lidar_centerpoint): added the cuda_blackboard to centerpoint #9453

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

feat(autoware_lidar_centerpoint): added the cuda_blackboard to centerpoint #9453

wants to merge 23 commits into from
+139 −87

Conversation

knzo25
Copy link
Contributor

@knzo25 knzo25 commented Nov 25, 2024 •
edited
Loading

Description

This is part of the series of PR related to #9722

List of PRs:

Depending on your machine and how many nodes are in a container, the following branch may also be required:
https://github.com/knzo25/launch_ros/tree/fix/load_composable_node
There seems to be a but in ROS where if you send too many services at once some will be lost and ros_launch can not handle that.

Related links

Parent Issue:

  • Link

How was this PR tested?

The sensing/perception pipeline was tested until centerpoint for TIER IV's taxi using the logging simulator.
The following tests were executed in a laptop equipped with a RTX 4060 (laptop) GPU and a Intel(R) Core(TM) Ultra 7 165H (22 cores)

Node / processing time [ms] Current PR
/sensing/lidar/top/crop_box_filter_self/debug/processing_time_ms 5.81 N/A
/sensing/lidar/top/crop_box_filter_mirror/debug/processing_time_ms 4.59 N/A
/sensing/lidar/top/distortion_corrector/debug/processing_time_ms 10.96 N/A
/sensing/lidar/top/ring_outlier_filter/debug/processing_time_ms 10.69 N/A
/sensing/lidar/top/cuda_organized_pointcloud_adapter/debug/processing_time_ms N/A 3.75
/sensing/lidar/top/cuda_pointcloud_preprocessor/debug/processing_time_ms N/A 1.00
/sensing/lidar/concatenate_data_synchronizer/debug/processing_time_ms 7.83 0.70
Total 38.8 5.45

Notes for reviewers

Note 1:
The main branch that I used for development is feat/cuda_acceleration_and_transport_layer2.
However, the changes were too big so I split the PRs. That being said, development, if any will still be on that branch (and then cherrypicked to the respective PRs), and the review changes will be cherrypicked into the development branch.

Note 2:
Now that the cuda blackboard itself has been added to autoware, this PR can be tested in a standalone fashion as well

Note 3:
Due to a missing feature in the blackboard, the following version is needed for now: autowarefoundation/cuda_blackboard#1
How that feature will be merged into autoware is explained in the respective PR

Interface changes

An additional topic is added to perform type negotiation:
Example: input/pointcloud -> input/pointcloud and input/pointcloud/cuda

Effects on system behavior

Enabling this preprocessing in the launchers should provide a much reduced latency and cpu usage (at the cost of a higher GPU usage)

@knzo25
feat: introduced the cuda transport layer (cuda blackboard) to center…
4686ce6 
…point

Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
chore: fixed compilation issue on pointpainting
46d5dcd 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@github-actions github-actions bot added component:perception Advanced sensor data processing and environment understanding. (auto-assigned) tag:require-cuda-build-and-test labels Nov 25, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 25, 2024 •
edited
Loading

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

This was referenced Nov 25, 2024
Merged
Closed
Open
@knzo25 knzo25 marked this pull request as ready for review November 25, 2024 09:46
@kminoda
Copy link
Contributor

kminoda commented Nov 26, 2024

@knzo25 Thank you for your great work 🎉

Do you have any documentation for the cuda_blackboard package? Just a simple API references and overall design in a readme would be helpful to review the PRs.

@knzo25
Copy link
Contributor Author

knzo25 commented Nov 26, 2024

@kminoda
Yes, sorry. I am adding some results (processing time, memory consumption, cpu use), documentation, etc as of now, but wanted to open the PRs before to the the ball moving

@knzo25 knzo25 mentioned this pull request Nov 26, 2024
Draft
@knzo25 knzo25 requested a review from amadeuszsz November 26, 2024 05:38
@knzo25
Copy link
Contributor Author

knzo25 commented Dec 13, 2024

@kminoda
Added the cuda_blackboard documentation in https://github.com/knzo25/cuda_blackboard.
As discussed internally, I will create an issue to manage and discuss about this series of PRs

@knzo25
Merge remote-tracking branch 'awf/main' into feat/cuda_blackboard_cen…
c2f4b56 
…terpoint

Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25 knzo25 mentioned this pull request Dec 23, 2024
Open
3 tasks
amadeuszsz
amadeuszsz requested changes Mar 4, 2025
View reviewed changes
Copy link
Contributor

@amadeuszsz amadeuszsz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvement, as cuda blackboard is 🚀! Apart from review comments, please take a look on pre-commit log (missing header).

@knzo25 knzo25 mentioned this pull request Mar 19, 2025
Open
@knzo25 knzo25 added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label Mar 19, 2025
@knzo25 knzo25 self-assigned this Mar 19, 2025
@codecov Codecov
Copy link

codecov bot commented Mar 19, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 0% with 33 lines in your changes missing coverage. Please review.

Project coverage is 25.93%. Comparing base (dbc5923) to head (53548ad).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ion/src/pointpainting_fusion/pointpainting_trt.cpp 0.00% 16 Missing ⚠️
...dar_centerpoint/lib/preprocess/voxel_generator.cpp 0.00% 7 Missing ⚠️
...rpoint/lib/preprocess/pointcloud_densification.cpp 0.00% 5 Missing ⚠️
perception/autoware_lidar_centerpoint/src/node.cpp 0.00% 3 Missing ⚠️
...autoware_lidar_centerpoint/lib/centerpoint_trt.cpp 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #9453      +/-   ##
==========================================
+ Coverage   25.92%   25.93%   +0.01%     
==========================================
  Files        1383     1383              
  Lines      106873   106810      -63     
  Branches    40931    40897      -34     
==========================================
- Hits        27708    27704       -4     
+ Misses      76459    76399      -60     
- Partials     2706     2707       +1
Flag Coverage Δ *Carryforward flag
differential 4.19% <ø> (?)
differential-cuda 6.41% <0.00%> (?)
total 25.94% <ø> (+0.02%) ⬆️ Carriedforward from e0bfd82

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@knzo25 knzo25 mentioned this pull request Mar 19, 2025
Merged
knzo25 added 5 commits March 19, 2025 15:28
@knzo25
fix: fixed standalone non-composed launcher
2bd110c 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
chore: ci/cd
3cd4c5e 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
Merge branch 'main' into feat/cuda_blackboard_centerpoint
68c61fd
@knzo25
chore: clang tidy related fix
ffbfb88 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
Merge branch 'feat/cuda_blackboard_centerpoint' of github.com:knzo25/…
9e96404 
…autoware.universe into feat/cuda_blackboard_centerpoint
amadeuszsz
amadeuszsz approved these changes Mar 21, 2025
View reviewed changes
Copy link
Contributor

@amadeuszsz amadeuszsz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for centerpoint!

@knzo25 knzo25 mentioned this pull request Mar 24, 2025
Merged
@knzo25
Copy link
Contributor Author

knzo25 commented Mar 24, 2025

This PR can be safely merged once autowarefoundation/autoware#5924 is merged and is tested with the perception evaluator

knzo25 added 4 commits March 24, 2025 09:33
@knzo25
chore: removed non applicable override (point painting does not suppo…
482cc16 
…rt the blackboard yet)

Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
chore: temporarily ignoring warning until pointpainting also supports…
373f2e1 
… the blackboard

Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
chore: ignoring spell
4dedbb3 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
Merge branch 'main' into feat/cuda_blackboard_centerpoint
39b12e4
@knzo25
Copy link
Contributor Author

knzo25 commented Mar 24, 2025 •
edited
Loading

Executing the evaluator here : TIERIV INTERNAL LINK

Update: tests pass

knzo25 added 5 commits March 25, 2025 11:48
@knzo25
feat: removed the deprecated compatible subs option in the constructor
55b08eb 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
Merge branch 'feat/cuda_blackboard_centerpoint' of github.com:knzo25/…
4f3e17d 
…autoware.universe into feat/cuda_blackboard_centerpoint
@knzo25
Merge branch 'main' into feat/cuda_blackboard_centerpoint
e0bfd82
@knzo25
chore: bump the cuda blackboard version in the build depends
74c884d 
Signed-off-by: Kenzo Lobos-Tsunekawa <kenzo.lobos@tier4.jp>
@knzo25
Merge branch 'feat/cuda_blackboard_centerpoint' of github.com:knzo25/…
53548ad 
…autoware.universe into feat/cuda_blackboard_centerpoint
@knzo25
Copy link
Contributor Author

knzo25 commented Mar 25, 2025

@YoshiRi @badai-nguyen @tzhong518
Tested both centerpoint and pointpainting and they are publishing detections. I think that should be enough test for this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kminoda kminoda kminoda left review comments

@amadeuszsz amadeuszsz amadeuszsz approved these changes

@badai-nguyen badai-nguyen Awaiting requested review from badai-nguyen badai-nguyen is a code owner

@ktro2828 ktro2828 Awaiting requested review from ktro2828 ktro2828 is a code owner

@miursh miursh Awaiting requested review from miursh

@tzhong518 tzhong518 Awaiting requested review from tzhong518 tzhong518 is a code owner

@YoshiRi YoshiRi Awaiting requested review from YoshiRi YoshiRi is a code owner

@yukkysaito yukkysaito Awaiting requested review from yukkysaito yukkysaito is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@knzo25 knzo25

Labels
component:perception Advanced sensor data processing and environment understanding. (auto-assigned) run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) tag:require-cuda-build-and-test
Projects
Software Working Group
Status: In Progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@knzo25 @kminoda @amadeuszsz